On the Relationship Between Binary Classification, Bipartite Ranking, and Binary Class Probability Estimation

We investigate the relationship between three fundamental problems in machinelearning: binary classification, bipartite ranking, and binary class probability esti-mation (CPE). It is known that a good binary CPE model can be used to obtain agood binary classification model (by thresholding at 0.5), and also to obtain a goodbipartite ranking model (by using the CPE model directly as a ranking model); itis also known that a binary classification model does not necessarily yield a CPEmodel. However, not much is known about other directions. Formally, these rela-tionships involve regret transfer bounds. In this paper, we introduce the notion ofweak regret transfer bounds, where the mapping needed to transform a model fromone problem to another depends on the underlying probability distribution (and inpractice, must be estimated from data). We then show that, in this weaker sense, agood bipartite ranking model can be used to construct a good classification model(by thresholding at a suitable point), and more surprisingly, also to construct agood binary CPE model (by calibrating the scores of the ranking model).